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Survey shows general: agreement: - 
on the meaning of “probable” and 

"some equivalents, elsewhere much 

" disugrecment, - 


| 


ot THE. DEFINITION OF SOME ESTIMATIVE . OS ark 
Bae EXPRESSIONS |— bat 


Finished intelligence, particularly. in: making estimative statements, 
uses a number of madifiers lke “hi ghly probable,” “unlikely,” “possible” 

- that can be thought of as expressing d range of odds or a mathematical 

‘probability, cand these are supplemented by various other expressions, 
especially verly forms, conveying the-senge of probability less directly<~ . Gs oo rae 
“may,” “could” “we beliave.”’ Certain other words express not proba- as 

: a bility but quantity, imprecisely but perhaps within definable ranges— 

foe OEY a tee. fee “few,” “several,” “considerable.” Some people ‘object, to any effort: to" ' “| 
< © define the odds or, quantities meant by such words. They argue that 


context always modifies the ‘meaning of words-and, more ‘broadly, that 
rigid definitions denrive language of the freedom to adapt to changing 
mecds, 

It is possible, however, to state the definitions in quantitative terms 
without making them artificially precise. And if two-thirds of the 
‘users and readers: of the’ we6rd probably, for example, feel it conveys 
a& range of odds between 6 and 8 out of 10, then it is more useful to 
give it Mis definition than to define it more or less tautologically in 

arms of other words of probability. This would not’ deny to context 
its proper role as the arbiter of value, but only limit the range of its 
influence. Nor would it freeze the language in perpetuity; as the 
moanings of the words evolved the quantitative ranges could be 
changed. a ; ne im 


An 


. This article describes the results of a survey undertaken to deter- 
ining if such words are indeed understood as measurable quantities 
8 2a. en. andif so to ascertain. the extent to which. there is-a-conseysus about: il.) & 
oS “the quantitative tange of each, A’ three-part questionaire on the : 
_ subject was distributed in the intelligence commiunity—-to INR/State, 
a Sue ete ce ale DIA- Office of Istimates, and five CIA offices—and a simplified. : 
“version. of it was-sent to policy staffs in the White House, State, and 
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the Pentagon. Responses were reccived from 240 intelligence eal 
and 63 policy officers. 


ea nn ata aaa aOR hoa mR RN 8 


The responses showed a satisfactory consensus with respect to 
various usages of likely and probable, phrases expressing greater ccr- 4 
. tainty than hese: and modifications of chance—good, better-than-even, 
slight. ‘There was no satisfactory ao eenee on the meaning of pos- pon 
sible or a wide varicty of verb forms such as we believe and might, 
«.... ‘There. was also littl’ agreement. onthe non-odds: quantitative Words. fore Re 
- a such.as few and many. - The policy offices consistently, assigned lower 9 

probabilities thari iitelligenée analysts’ did. Correlation between val- ° 

“ues assigned in and out of context was good. 


The Questionnaire 


-. Part Oe of.the it estidnnaire ‘listed: 41 expréssions that: anight ‘be 
eer re “thought oF as indie: ating odds ‘and off ved the choice of 0, 10, 90, ne 


“pitied ty cael ft the’ o eporident believed “tit no aantilative 

“answer was satis factory he could mack “Not Applic: able” instead, 

- : * . 1. These expressions.of course had’ to be: judged without benéfit of ‘con- ° 
, text, but in-order to check on the validioy of such judeients some of 


‘them were repeated in Part Two, where they were included in 17 
sentences taken fron intelligence documents which had been produced 

_ in six dilferent-alliccs of the community. :-The names of all persons: 
and countries in the sentences were changed to sterilize them against 
bias. Part Three then listed nine expressions of magnitude not refer- 
ring to probability and offered an assortment of ranges for each. 

j The idea of a si nines relative, but for. purposes of Parts One 

; and Two it was defined as requiring 70% or more of respondents to 
name orids w iehi n iS points, plus or minus, of the most frequent - 
response. [f the odes or chances most frequenily. specified for pos- “4 
sibly were 50 out of a hundred (as they were) and 70% of all the 
responses had fallen within the range 40 to 60, the requirements for 
& conscnsus on this word would fave. -been: satisfied. Only one figure Cita} bate 

“was recorded’ for’ cach questions: ‘when’ an answer. was ranged ‘by a5 
marking | several adjacent figures, it was recorded as the mean.’ Mr. : 
Kent's range of UW) to SO for possible would thus have been recorded | 

‘ _ > -as.60. Definitions were aso considered invalidated by 20% or more , 

_ of “Not Applicable” responses rejecting the question. 7 ; 
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“The replies were tabulated in fou? categories in n desonding order 
of valid definition, as follows: : 
Category A—-a consensus including 90% or more of. all ree 
H spondents. : : 
Category B-—a consensus including 70! % to. 89% of all re- 
spondents. . 
Category C—no consensus, but fewer’ fae 20% of respondents : 
/ marked “Not Applicable.” 
» Category, Dna. consensus, and 20%: .or more.. of tespondents 
: “marked “Not Applicable” ~ 


eoriree = oat 


t 
s 


The following Eales summarize the findings of the survey. After 
each expression from Farts Que and Two are shown the’ odds mast 
fre Savi: specified and the pe reentage of respondents within 10 points 

= of that. i ‘or questions, submitte d to policy officers: “as We ell.as analysts, 
their Fesponses are shown separ ately, The expressions of magwitude 
in Part Three. are listed’ with the percentage of - “Not Applicable 
“Yi, > responises and the. most frequent response: for: each. : ae eho 
OF the 41 expressions in Part. One threc fell into Category A fe uper- 
.. consensus); thirteen into Gategory B ¢ consensus), seventeen into Cate-. 
gory C (no consensus), and cioht into Catecory D (refeeted as inde- 
finable), From Part Two five expressions in context fell into Category 
f B, twelve into Category C, and three into Category D. All the 
quantitative phrases in Part Three.were rejecte d as not measurable 


i _-. , by 20% or more of the respondents except for next few years and 
PO next year or so. Though rejected by only 7%, next few years found 
i no consensus: 19% marked 2 to 3 years, 30% 2 to 4 years, and 34% 
i 2 to 5 years. Next year or so meant 1 to 2 yor to two-thirds of ie 
: respondents, 1 to 3 years to the rest. 


PART ONE (No Context) 


Opns—Mosy Fre- Prercenr AGREEING 
Expression : : QuENT Response = Wrrnrin 10 Porrrs 


Analyst - Policy : Analyst. Policy 
lacs A eee 100% ae 


- 99% - 94% 
2 + 06% 92%, 
vt : “PGE: 01H: 
mat : ale 
Probialily ste.t cute cdlnehcgtea wageee nots 18 70, 90% 86% 
Probably not .-...... isd Beatie sean 20 20 85% . 16% 
Probably will 00.0020... eee eevee de. BO —- 85% 


yo 


fe é Lt pee ashe ee Mg ee ae So teenpe neta meat conan tarmst 
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Opns——Mosr Fre- . PEncENT AGREEING 


a tt a om eetattanmaatitatth Nee At 


; ; JEXPRESSION ; QuENT Response 9 Werntn 10 Pornts 
a i. Analyst Policy Analyst Policy 
fe Category B (70%-89% Consensus) (Continued) |: 
* psf : Highly probable ....6........000. Sees 90 ‘85. 83% . 87S L 
1 a Likely wove. ee. dee ahi ee i Pehnwedes TO. 83% — ' 
H ~ Undoubtedly .t......... te Mokat Cesc d wie 100 90: 81% 86% a 
Good chance ...... eis: se eee ge dre http . T° TO 81% 81% Sua sted % 
oe, Highly Jikely oc ec gece i Adi dee ee  $90 BO 80%" BIG 
|S wget s “Anlikely se ad oGa Gentes oaenuatieg Hoenn Oe a ©. 20 "80% ©7195". Bae 
Seems likely 20 ......0.0. Tp atinain Seated eee 100 me 80H 
: Better than even chance 2.0.0.5 ...00..00005 60 . 60° 78% 87% | 
: Soino slight chance ........ tial rasta ths iat .. 10 10 11% 19% 
es te sop ae ee Ria oe gives aedes 50, ee, 89a. 
; Category C (No Consensus} 2 
Scents Unlibely cco tine kee ap tae gt GORD. AA Ai BIB og. ee 
- Might: Cee eee eae BO. «BO 66% = BNS% 
May i: 50 00 6650 
Could be oe ted i 60. me O5% pee ‘ Peeyee | 
‘.ulixpect toad Pe! a BAG 
~ Could’ 50 60%: 56%  - 
SC NEUSE onc te Be dak Pe ee ot ick areas hee oe _, 59% 2 
“Evidently ole... a | ( n° ms 
Apparently ccd oun pala wa da dape path Hala asd cea . 7% —_ ' 58% = 
SUUHOS SE cd ey pets Sostects cohaudele tee cavidtens 60 -~ 58% = — 
Belevés yn. ae is a ea oad 70 70 55% 54% -. a 
Should o........0.. es odie 9 El aesitends 70 — 54% — 1 ded 
POs istic aot etna sd 99 ete es BOL S00 B8% OST 
“Might ve expected Niche tuscan at Mec, he 50* — 51% 
lidieatis HES Adanchidie sea once todas - 70 _ - 81% — a 
Might be anticipated .7.....0......0 0005 50” OBO 56% 50% 
Apparentiy is intent ...... Win Pega oa atts 60* _ 50% — 
“Serious possibility ..0.0.00 1... cc cee, 60* = 70 49% 55% 
Estimate oo... ..0000. : fi 70 56% 57% 
SeGins® ¢ kasi ied fh gadiee ee dich ed seme Nee dace — 55% —_— - 
OUGHE Cie anes ete Mote acne ea cae — AIG : 
Neel) 3 gidaue fewsiar edie eset ues -—- 35% | 
Reportedly -* 50 35% 52% 6 casts sat 
_. Somewhat ie 21% meee te rae 
& a, Ostensibly :. Sh i 20%. — note es OO 


* The. most frequent response to these questions was of the same -order as a 
second most frequent. Graphed, the responses would show two frequency peaks, 
forming, in statistical terminology, a “bimodal” curve. 

t The curve for this response is trimodal. 
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- : yee ie 
ug Peak i ‘PART ‘TWO (In Context) - 
: pe ; 
. : EXPRESSION : F : : Opps—Mosr Fne- _ Pencent AGREEING 
: | (IN CONDENSED CONTEXT) vo. gent Resronse |. Witutn 10 Ports 
wow ~~ | _ Analyst” Policy. *, “Analyst Policy 
‘Category’ B ‘(70-89% Consensus) . ; 
4 ‘We believe the chances are good that... 0°70. 0° 86% — 
= » We believe’, . . will. not bes. 6.20.0, 8080) 76% 63% 
i * Undoubtedly, ..... will not be. o. ...... 100 © sgcceae 76% 
a FS We estimate . . . will not-be.. .. 20.08... 80 °° 70 -. T4A%. 70% 
Barring +... the economy will probably PA, ee ue a, po. 
, oF eg -, continue... en || a OTe BS oa Wesson cat oe 
be Per nea te eo gece: Cntepprye (No Consensus) ee e ; 
; Apparenily, . . . will not be . case. TO7 me + 68% = - ae 
If... continue... , the president might wi ee 3 
Bia So DO MANN: WSF No yo okies oo : 50. . 50 (65%: = 54 
- might also-take-... action. ..... 50 62% 
». . references ... to undiminished im- ‘ ; 
/ | portance othe Ss suggest a belief ; 60* — 59 G6 ms! 
At is possible that. >. will become... 2. 50.) 0 BO 58%: 57% 
: e Sv... visit... indicates that se ds be : ee : nae 
oe Dae Hane Cet Pocts Ae eadniias a 
fb nS "s+ a ee Visit suggests “.. . progress tet BO ee ee ee nee 
_ Ten Vos We: beliewe 20... ‘there is a possibility ¢ "7 eee en ee ds 
PTE Se MAb ian ater siete late eo ans, BOG? Boe © 8 pe, oo 
* ¢ » speech . . ..conneyed the impression ; ; 
WBE ea aEP* x Aah Ae ceed ok Ale cas 8 GO* Po 46% — 
» commenls suggest . . . changes may 
twell be less than speech . . . might indi- 
CME eg a ho Bog etal Shane ad scers eats 70* G65 43% 40% 
an 7 Category D (Rejected) to 
: + Comments suggest .., that... .gov- i ; 
: ernment is notcommitted .. 74 ....0.... ~~ OF 50t 185% ° 235% 
: i ; *The full context on these questions was the sentence, “Although Jacking the 
: drama of visits by top leaders, the travel of these delegations to Albania indicates 
that the momentum of the Alhaniin-Pelish rapprochement is being raaintaiued and 
suggests that some progress. is being made in reducing the area of remaining 
: : ideological differences.” Respondents were asked to specify the probability that ; 
Albania and Poland were headed toward a rapprochement and the probability that : 
| the ideological differences would bo settled, : 
* Respondents were esked for the probability that the speaker believed what 
r os he conveyed. : Le oh oo = Binet Ae Bes ve 
i | . * Respondents were asked for the ‘probability that changes would be minor. 
[: "Sitng «fe 4 Respondents were’ asked for the: probability of that’ to’ which’ the “sovernment 
is not committed.” The full context is given on page 73, . : 
Ry eee * Bimodal. ; 
f vee” £ Primodal. 
oot ae a 
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-Estimative Expressions 


EXPRESSION 
. (IN CONDENSED ConTr xr) 


‘Opps—Most Fre-  PEencent AGREEING 


i 
he 
os hie 
| 
i Quent Resronse = Wirurn 10 Tornrs 


Analyst Policy’ | Analyst Policy = 
‘Category D { Rejected ) (Continucd).” ee 
This raises the question whether . . . they 


TLE pei toe Ei des as Ge itera ae at 51% — 
We do not expect them to change .. & wees OOF OT 22% — 
* Cuba has allegedly bought... 00.05.0... 50 — 38% — 


_ ®Phis question was-a non-sequitur. , The full context is given ow page 73., Ae : 
“7 # Trimodale's.- Se RS PET as Ne ee 


wd 


PART THREE (Words of Magnitude)” = * ae 
: - Percent = Mosz Fre- 


-., - EXPRESSION i ; a - REJECTING QUENT Rusponse . 
Considerable Meee ; Clashes ast aunt a eee ye ; 10-100 
UIDs Gen oes asic iets oarseeteaciea NADIR «» 10-1000 , 
74 Substantial, {portion )/ 202. ieee Ps ae 1 BO% 20 250%: * 
Significant (portion) 7.00200... DANG mie As S19. "20-50%. 


‘Limited (portion) 


a “Several eee ace’ " 2-5 

; Few .2 00... Lees athe oe oe Pitch, Bein tee oe © Qa 

“Next few years-o..-.. chee eee Sg Pins ae oate 7 ne a5 years 
Next year or so... eae fh 1-2 vears 


‘The difference between the good consensus on a set of odds for 
one expression and no consensus on another shows-up clearly when 
the odds are graphed aceording to how frequently each set was 


 speéified in’ thé responses to a quéstion. When 709% of all responses 


: E 
‘fall within 10 points of the most frequent one, the graph has a steep 


curve and a narrow base, The high, narraw peak indicates a clearly 
defined conscnsus, whereas.a broad-based curve with a single peak 
shows less agreement and a curve with several peaks reflects clear 
differences about what the word means. 

Steady retrogression from consensis can be secn in graphs of sample 
responses from. successive categories. Following are these seven from 
Parts One and Two. 

i : : ’ Carecory . 
- Out of context: ; oe 


: ots : Tle MMe 8 San ae aes 
rir Almost Certainly”. 1.0.2... he ‘ TOA 
Probably 0.0... 0... es, Sted Seeds Babe tied aacb egies B- 
Possibly... 0. 00.0000. oot bev fea . oC 
Scrious Possibility Shs Atte cag als Gea pa ean SO erence tnd caw Cc 
. Seems 2.002... Bap a webinar Teles ie als cats cate vd uns a tannic dicesacl D 
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Carecony “| 
In context: 


continue provocations. against South Korea? ...... 0... vec D 
“At the same time, the reservations conveyed in the military 
i commnent suggest that the practical military changes resulting | 
i from the new line may well be: less dramatic than the tone of 
» y i . 
} de Gaulle’s speech might indicate—and that in any event) his 
i : government is not comunitting itself to a one-weapon system | oe | 
i “. of defense” expresses what probability that the military will hae ae isha g : i 
; “have a .oné-Wweapon system? voy yel Chien Satie DO BO : 
Ht : as : ts: 2 : ee z : é : Bt Sis Lore i 
The.red line in each graph traces the response pattern of 239 analysts, ~ <p 
the black line in the first four that of 63 policy officers. The dotted 
black line is the latter adjusted to scale. “Mode” designates .the 
<< _ peaks of most frequent response. , : aoe 
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se 
the basc of the curve 
range Ueloven To and 9a, 


Analyst Curve 
Black Line e+ Pabicy Curve 
Gottest bine -- Pulacy Curve ad- 
: Bee JUsted tu Scale * * 


“* RESPONOENTS 


oops OT OF 100 


and No. 1, Category A: Almost Certainly (sige Range ae) 
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: Estimative Selanne 7 Ere eS arbameiee as 


Bj. eae EE ee Bars eee dp stir Ge patties peer eet 
7 Ronponacs on "Probably? ahow tro “dufinite * 
odes, 70 for policy and a) for analyst, the 
base ip almost twice as bread as Graph i's 
and the aldes ere not an mtuep. 


4 KEY 
Hos-tee--+ Red Line --+-- Analyst Curve 
Black Line --~ Policy Curve 
butted Line -- Pulicy Curve Ad- 
justed to Scale 


OvbSs our cr 100 


Red Line 
Biack Lin a 

Shas Dotted Line” aaa copoly Care e Ad- 7: 
: ! justed to Scale 


ODDS OUT OF 165 


“Grari No. 3. Category C: Possibly (Significant Range 10-60). 2s 
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"Red Ling ---- Analyst Curve 
Black Line =-~ Polley Curva 
:_. Dotted Line -- Policy Curve ad- 


dusted ta Scale 


The curve for ‘Serious Possibility” has. 
L.. three distinct’ modes and e very broad base 
... Fanging from 25 ta 93, 2 
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_. Grapu, No. 4. Category C: Serious Possibility. (Sig 
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Seems. (Significant Range 30-80). 
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Estimative eee a labile 
; 4 rs . % 
‘ 
1) ; ae on pe eng es ca etch my acme! 
es ‘ 4 i : 
—_ ee ge ; he : 
: This question wag included of the anstyst . aes é cone oa 
4o + version only, The’curve ranges trom @ tu LOO * gv Pee SE agi ara ie Ber 
and shows no clearly delined mode. There is 7 
no consensus fur the statement vf probatillty. - 
‘ ioe Beet, “oxey . oe , 
aS ‘ Red Line =~ Analyst Curve . ‘ 
| ‘ ; ¢ ; “ ‘ 
: ; ~“ : 
ae a bg ra : 
: : | 
be 2 a rf 
w rr rn ‘0 “eG Ta. 7 me oe ie ae 
he as Batt aa Tass SS ees Opps oy OF Too te ss Eg ORs : 
a Grarn No. 6. Category D: Korean Question (Signa Range 5-85). 
Ua this coaplacamet ques 
curve ranges eae a to tog aud ‘ ot 
defined mode, There is ho consensus tor tie 4 
gtatement of probability. 1 j 
BG DEAS Edens dee hens hea sed oe os ee 
. ey : & ees I 
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‘ fs 7 
| a 
i : 
i 7 : 
t 
i . deg Wt 3p j % 
Is oa 
# me gre. 
fy Fay ca Aes a 
_ opis our OF 100 
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~ Conclusions a | . 1 ee eM E* 
Of the 303 questionnaires returned, only one indicated that no 
quantitative equivalent was suitable for any of the probabilistic .ex- 
pressions. All others selected sets of odds for at least half of those _ 
listed in Part One, and .80% did so for ‘two-thirds of them. Even : 
. though a number who disapprove of quantitative definitions probably as 
just did not bother to return their questionnaires, the results appear oe 
to indicate that the vast majority in the intelligence community con- | 
_-Sider.it legitimate to think of suéh expressions in quantitative terms, ~ 
es, oe © On-the other hand, although more than 70% of both analysts and . : 
, policy offivers agreed within a 20-point range on the expressions in aa i 


_ Categories'A and B, the results for some offices on the analytical side 
did’ vot egree with the consensus for all ‘analysts, and there.were - ; 
similar execptions. among the policy offices. So.when an analyst i oe 

; one office uses the word prebably, policy officers and analysts in other: 162-74... 

Pag a tt co offices’ do ‘not necessarily interpret-the word: to mean the.same thing, “00 27 

ws In Categories A and B, however, the differences are usually not great, — j 


ty There. follows the. quantitative. definition—most frequent plus: and’ 
minus 10—of expressions on which there was found to be a satis-_ 
. E factory consensus. « «| Be oe, See rae 
oe ee 2 Part ee TR opie “ Cancers 
: ‘our oF 100 
Ca Ie aC RE Pr ren an Ctapatecal whe ier one 2 a -. 90-100 j 
WALES 2 Setencnanl rd pai et ns al Banat ous ait lo Pi gecg tied ehh 90-100 ‘ 
Ahnost Certainly 20.0000 ee. Lieade sched 80-100 i 
Undoubiedly 3s Pebdian slates a ae a 210 SS 106 SO 
po EVN a cc rhes MMe Bete tls muita Sietl, tren tpteet., 75-95 ; 
Highly Probable o.0.0...0.0......, See Tiina ta edt a Pas aye ve . 95 . an 
Probably Will 2... 2, eee Senet obe eka Ata tons SE 70-90 ; 
: Becbably2. ie teel, bes EW oe 2a ae Sigel eto te ~.. 60-80 
PEM oy aa Baan aot ee PR rnd ae hee flats leh thinbd 60-80 
Good Chance .......0... 00.....0..., deMehauce ge baile wt 60-80 
ONG TAINS pase Segara Geld chen cet tana ue yg det 60--S0 
Dotter Than ven Chance . peste shite oaks Fc da net iante, SUGE Le _ 50-70 
May. cee net wees dip Fae athe tetvahale ee eleteoula ue’ 40-60 
Probably Not .).....00...0000., Ga appr tatag a ote ene 10-30 : 
REY saat eee whe cece gers), csga yet ee et 10-30 : 
Some Slight Chance .........05..,,.. Rit osc ccknm teats, wena a 0-20 a 
e ac%t., .,. The out-of-context, definitions in Part One’ were spot-checked by 1 
Foo" ") “the seritence: questions of Part Two, . The results are not conclusive: 
only one sentence was provided for context, and there was no way 7 
i of telling if respondents were influenced by personal knowledge of { 
* . the subject matter. But despite: these limitations, because the most . | 
i a co gg” = ee vee | < % 
+. : : 4 : ‘ * 7 ¢ 
Se ee oe Rates LR 70008-0 
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Estimative Expressions. _, thet : - mmniichialimD 


frequent definitions in and out of context agrecd within 10. poitits, . 
it appears that nearly the same. meanings were conveyed either way, 
The comparison appears below. - ake 


MOST FREQUENT RESPONSE. 


. In Conrexr ALONE - 


; ; Analyst Policy . - : Analyst - Policy 
Undoubtedly... sae TOO SS Uadoubtedly-2..,. ... 100 (0 ‘: 
aa ee foe Believe... Sos 7. 70 ws 
- +. .Apparently 200... Te me oP Estimate oH ie oa ea ic) are 70 Sing eile e Be ane 

| Indicates that 2.0077.. 0 7g) LE” Apparently" wo... aoe, ee 

Believe the chances are- Indicates. that pare aaa a Se i 
good ........, poead. eOe -~ _ | Good chance... Spee 70 70 L 
Possible 20.202 .00. 00, 50 OBO Possibly... 2.2... Pree ~ 50° BO 

Might) ....000.. etka 50> as Might 2.1 ..... Sakae 50 50 


Although the coupling of a verb of opinion with an expression of , 
VP gets edds,.as. in “WW. e.believe the chances are good,” seemed. not to affect} "- ee ee 
" ‘the meaning of the latter for the respondents to the survey questions, 
this writer agrees with Mr. Kent's purist that the doubling up of 
ris’. .probabilistic words is potentially confusing and should be -avoided. 


: | a The response pattern on the Korean question (page 73 and. Graph 6). } 
sg hE has an’ interesting side light in‘that the probalility queried does not ' 
follow from the estimative sentence, The cpicstlannaire wus not 
designed to test the propensity of analysts and policy oficers to draw . 
unsubstantiated conclusions, but in this. one instance’ only 35% of 
the respondents showed they recognized the non-sequitur by mark- 
ing “Not Applicable.” : sesh oes FS 
_ The survey showed that for expressions on which there was a 
consensus (and some others) the most frequent response was the 
same from policy officers as from analysts within 10 points plas or 
minus, “But whore diffcrences did occur, the policy definitions were 
consistently on the conservative side; see the following examples. 
MOST FREQUENT RESPONSE ; 
; Analyst Policy 
Sei tated in de tytn eaten: lee... 100 90 " 
: 90 8&5 2 
90 &0 : 
; 75 70 +. aghcst 
tea, wea silty TE CLIO, > eta 
howed there is little consensus on: 
the eommon expressions of vague magnitude, at least without the 
a. guidance of context, - sp Aaa aot ; . ; 
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| TT a eee Estiniative Expressions . 


‘Faults in-the Survey’ fe il . 
An effort was made to keep the questionnaire as simple to under- 

~ stand and as short as possible, “In Parts One and Three the effort 

_ Was generally snecessful, but: Part Two was neither simple nor short. 
“Most of the questions in the Jatter related to specific people and 
places, and there was: danger that respondents would permit their 
opinions and knowledge of the. subject to influence their answers, 
In addition, several of the.estimative sentences were long and involved, 
carrying the hazard of confusion about.what they meant and what 

“ ‘was'wanted in évaluation of them. ... eae a ae een 
"For pragmatic reasons, administration. of. the survey ‘had to be- 
informal. It is possible that such things as attitudes of supervisors, 
officé collusion, or misunderstanding of the. purpose of the survey... 
could have introduced bias. A careful perusal of each of the ques~ 
fionnaires: failed-to turn up any obvious evidence that such factors 
influenced the findings. But if it were-done again the gtiestionnaire .. _ 


1 Pe mes TB a 4 wt : Hay Faye a Po. rs ae 
.;, should bo modified in ‘Part Two and “the conditions: under Avhich it is 
filled out should be controlled and standardized, ee ; se 8 
ops ’ ence, “te a 
a oe x, : 
aes . rt is an 
fhe 4 80: 0 ; CQMMTTAL | 

a ‘ a. i 
: se 7 2 oe 

gs oe ” : 
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